(Not Applicable)
1. Technical Field
This invention relates to the field of content-based image retrieval and more particularly to a method and apparatus for indexing and retrieving manufacturing-specific digital imagery based upon image content.
2. Description of the Related Art
Images are being generated at an ever-increasing rate by sources such as defense and civilian satellites, military reconnaissance and survellance flights, fingerprinting and mug-shot-capturing devices, scientific experimentation, biomedical imaging, architectural and engineering design, and industrial manufacturing. In consequence, image knowledge structure and an access method for retrieving images are two significant problems in the design of large intelligent image database systems. Content-based image retrieval [CBIR] represents a promising and cutting-edge technology useful in addressing the problem of high-speed image storage and retrieval. Specifically, CBIR refers to techniques used to index and retrieve images from databases based on their pictorial content.
Typically, pictorial content is defined by a set of features extracted from an image that describes the color, texture, and/or shape of the entire image or of specific objects. This feature description is used in CBIR to index a database through various means such as distance-based techniques, rule-based decision making, and fuzzy inferencing. Yet, to date, no significant work has been accomplished to apply these technologies to the manufacturing environment. Notwithstanding, imagery collected from the manufacturing processes have unique characteristics that can be taken advantage of in developing a manufacturing-specific CBIR approach.
The manufacturing environment represents an application area where CBIR technologies have not been extensively studied. The low cost of computer systems, memory, and storage media have resulted in manufacturers collecting and storing more information about the manufacturing process. Much of the data being stored is product imagery collected from automated inspection tools. This imagery contains an historical record of manufacturing events that cause a reduction in product quality. Under the proper circumstances this data can be used to rapidly source product quality issues and improve product yield.
Semiconductor manufacturing is representative of an industry that has a mature computer vision component for the inspection of product. Digital imagery for failure analysis is generated between process steps from optical microscopy and laser scattering systems and from confocal, SEM, atomic force microscope and focused ion beam imaging modalities. This data is maintained in a yield management database and used by fabrication engineers to diagnose and source manufacturing problems, verify human assertions regarding the state of the manufacturing process, and to train inexperienced personnel on the wide variety of failure mechanisms observed. Yet, the semiconductor industry currently has no direct means of searching the yield management database using image-based queries. The ability to query the fabrication process is based primarily on date, lot, and wafer identification number. Although this approach can be useful, it limits the user""s ability to quickly locate historical information. For example, if SEM review has determined that a particular defect or pattern problem exists on a wafer, the yield engineer must query on dates, lots, and wafers to find similar historical instances. Although roughly 70% of all space occupied in the database consists of imagery, queries to locate imagery are manual, indirect, tedious, and inefficient. Therefore, this becomes an iterative and slow process that can prove unwieldy in the modern semiconductor environment where a single manufacturing campus having multiple fabrication facilities at one site can generate thousands of images daily. If a query method can be designed that allows the user to look for similar informational content, a faster and more focused result can be achieved. A process for locating similar imagery based on image content, for example the image structure rather than lot number, wafer identification, and date, would result in a reduced time-to-source. Hence, what is needed is a method for manufacturing-specific CBIR that addresses defect analysis, product quality control, and process understanding in the manufacturing environment.
A method and apparatus for indexing and retrieving manufacturing-specific digital imagery based on image content in accordance with the inventive arrangement satisfies the long-felt need of the prior art by providing manufacturing-specific, context based image retrieval in an industrial environment. In response to an industrial event, the inventive method can afford fast access to historical image-based records of similar industrial events so that a corrective action can be quickly taken. Thus, the inventive arrangements provide a method and apparatus for employing an image-based query-by-example method to locate and retrieve similar imagry from a database of digital imagery. The inventive arrangements have advantages over all content-based image retrieval systems, and provides a novel and nonobvious system and method for indexing and storing content-based, manufacturing-specific digital imagery for subsequent fast retrieval.
A method for indexing and retrieving manufacturing-specific digital images based on image content comprises three steps. First, at least one feature vector can be extracted from a manufacturing-specific digital image stored in an image database. In particular, each extracted feature vector corresponds to a particular characteristic of the manufacturing-specific digital image. The extracting step can comprise extracting three independent feature vectors, the three independent feature vectors corresponding to a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic, respectively. Notably, the extracting step includes generating a defect mask using a detection process selected from the group consisting of thresholding the manufacturing-specific digital image, comparing the manufacturing-specific digital image with a golden template, and comparing the manufacturing-specific digital image with a digital image of a neighboring product; and, extracting a feature vector for the substrate/background characteristic or the anomaly/defect characteristic using the defect mask. Moreover, the extracting step can comprise the steps of: distinguishing a defect-region from a non-defect region in the manufacturing-specific digital image; rendering the defect-region similar to the non-defect region based on an estimate derived from a region surrounding the defect-region, the removal of which forms a modified manufacturing-specific digital image; and, extracting a feature vector corresponding to the substrate/background characteristic from the modified manufacturing-specific digital image.
Second, using an unsupervised clustering method, each extracted feature vector can be indexed in a hierarchical search tree. Specifically, the hierarchical search tree includes data encapsulating nodes. The nodes can be leaf nodes or level nodes. Each leaf node encapsulates a feature vector. In contrast, each level node references at least one additional node. Moreover, each level node encapsulates a vector average defined by the average value of all feature vectors and vector averages encapsulated by nodes referenced by the level node.
The using step can comprise the steps of: forming a subset of feature vectors, the subset including at least one feature vector; establishing a main branch in the hierarchical search tree, the main branch having at least one leaf node encapsulating a feature vector included in the subset of feature vectors; and, adding leaf nodes to the hierarchical search tree using a top-down algorithm. In the adding step, each additional leaf node encapsulates a feature vector exclusive of the subset of feature vectors.
Third, a manufacturing-specific digital image associated with a feature vector stored in the hierarchicial search tree can be retrieved, wherein the manufacturing-specific digital image has image content comparably related to the image content of the query image. More particularly, the retrieving step includes several steps. First, the query image is converted into at least one query vector corresponding to a particular characteristic of the manufacturing image. Subsequently, a first-level data reduction of feature vectors stored in the hierarchical data structure can be performed based upon the query vector. The first-level data reduction preferably constructs a subset of the feature vectors comparable to the query vector.
Thereafter, relevance feedback can be accepted. Specifically, the relevance feedback can include a user-chosen selection of manufacturing-specific digital images corresponding to the subset of feature vectors. For each manufacturing-specific digital image in the selection, three independent feature vectors of manufacturing-based digital imagery can be extracted, the three independent feature vectors corresponding to a digital image modality and overall characteristic, a substrate/background characteristic, and an anomaly/defect characteristic, respectively. From the selection, at least one prototype vector can be calculated, the prototype vector corresponding to the particular characteristic of the manufacturing-specific digital image. Specifically, where three independent feature vectors of manufacturing-based digital imagery are extracted, each independent feature vector for each manufacturing-specific digital image in the selection can be logically combined, the logical combination forming the prototype vector for each independent feature vector.
Finally, a second-level data reduction based upon the prototype vector can be performed, the second-level data reduction resuting in a subset of the feature vectors comparable to the prototype vector, and further comparable to the query vector. Still, the retrieving step can further comprise the step of fetching from the image database a manufacturing-specific digital image defined by an intersection of the three independent feature vectors corresponding to the prototype vector.
In addition to the three steps, the inventive method can include a fourth step. The fourth step can include managing the hierarchical search tree. In particular, the managing step comprises the steps of: identifying a level node referencing redundant nodes having redundant feature vectors; equating the vector average encapsulated by the level node with all feature vectors and vector averages encapsulated by nodes referenced by the level node; and, purging the hierarchical search tree of the redundant nodes referenced by the level node.